feat(sync): add Novita AI model catalog sync by Alex-yang00 · Pull Request #7050 · anomalyco/models.dev

Alex-yang00 · 2026-09-14T10:08:17Z

Summary

Add a Novita AI model catalog sync provider
Fetch model inventory from https://api.novita.ai/openai/v1/models
Preserve existing hand-authored metadata and model inheritance
Keep existing entries when the API response is account-scoped
Skip remote-only models because the endpoint does not provide complete catalog metadata
Wire NOVITA_API_KEY into the hourly sync workflow
Add focused parser, API, registration, and retention tests

Data source

Novita model list: https://novita.ai/docs/api-reference/model-apis-llm-list-models

Required repository secret

The sync job requires the NOVITA_API_KEY repository secret.

Validation

bun test packages/core/test/novita-ai.test.ts — 9 passed
bun validate — passed
git diff --check — passed
bun test packages/core/test/sync.test.ts — 193 passed, 2 unrelated existing failures (Hyper and DeepInfra expectations)

Notes

This is intentionally a conservative first implementation: it does not delete existing model files missing from the API response and does not create incomplete TOMLs for remote-only IDs.

github-actions · 2026-09-18T05:47:44Z

Action items

[high] [violation] packages/core/src/sync/providers/novita-ai.ts:143 - Check: Capability flags must not treat a present-but-incomplete features list as authoritative false. Why: features?.has("reasoning"|"function-calling"|"structured-outputs") returns false for an empty or partial features array, and false ?? resolved never falls back, so existing reasoners/tool models can be rewritten to reasoning = false / tool_call = false and lose reasoning_options on the next hourly sync. The unit test only covers features === undefined. Action: Treat capability inheritance as “API claims only when the feature key is present in a non-empty features set” (or require explicit true/false fields); never map absence inside a partial list to false over resolved/authored values.
[high] [violation] packages/core/src/sync/providers/novita-ai.ts:1548 - Check: missingModelID must mark only skips that need manual catalog work, not every untranslated remote ID. Why: missingModelID always returns model.id, so non-LLM rows (now parseable with context_size: 0), unpriced IDs, and unverified reasoners all enter the missing-model issue flow and can never be intentional silent skips. That will spam [missing-model] novita-ai: … issues and retain any local file that collides with those IDs. Action: Return an ID only for skips that need lab metadata / verified reasoning controls; intentionally ignore image/embedding/other non-chat catalog junk (and document the filter in sync.md).
[high] [possible mistake] providers/novita-ai/models/qwen/qwen3-max.toml:1 - Check: Provider reasoning / reasoning_options must match this host and the lab baseline, not invent always-on reasoning. Why: Lab models/alibaba/qwen3-max.toml and first-party Alibaba mark reasoning = false, but this PR flips the Novita entry to reasoning = true with reasoning_options = [] (same pattern on qwen3-next-80b-a3b-instruct). On a multi-lab relay, [] means “no caller control,” not “uncertain,” and contradicts the lab non-reasoner baseline unless Novita truly forces hidden CoT. Action: Verify on Novita chat/completions whether these IDs actually emit reasoning tokens/content; if not, keep reasoning = false and drop reasoning_options; if yes and controllable, author the real wire control (with leading comment), not [].
[medium] [possible mistake] providers/novita-ai/models/deepseek/deepseek-v4.1-flash.toml:1 - Check: Relay reasoning options must start from the lab/same-surface peer set for that model. Why: New V4.1 Flash is authored as toggle-only, while DeepSeek/OpenRouter peers for V4.1 Flash expose graded reasoning_effort (low/high/max or equivalent) in addition to on/off. The header only documents toggle. Action: Live-check whether Novita accepts effective reasoning_effort (or equivalent) for deepseek/deepseek-v4.1-flash and other V4 Flash IDs; if yes, add the real effort values; if no, keep toggle-only and note that effort is not exposed on this host.
[medium] [violation] providers/novita-ai/models/deepseek/deepseek-v3.2.toml:1 - Check: Every toggle needs a leading top-of-file wire-path comment. Why: Sync rewrites many toggle reasoners (deepseek-v3.2, kimi-k2.5, qwen3.5-*, glm-4.7-flash, …) with [[reasoning_options]] type = "toggle" but only emits the thinking.type header for VERIFIED_THINKING_TOGGLE IDs. AGENTS.md requires the exact wire path above the first key for every toggle. Action: Emit/preserve a leading # Toggle: … comment for every Novita model that keeps or gains a toggle (reuse the verified thinking.type path where that is the control).

github-actions · 2026-09-18T06:10:08Z

Action items

[high] [violation] providers/novita-ai/models/deepseek/deepseek-v3.2.toml:1 - Check: Every toggle reasoning control needs a leading top-of-file wire-path comment. Why: The PR rewrites many existing Novita reasoners to [[reasoning_options]] type = "toggle" (and keeps toggle+effort on DeepSeek V4) without any leading # Toggle: … comment. Sync only emits VERIFIED_TOGGLE_HEADER for the hard-coded VERIFIED_THINKING_TOGGLE IDs, so preserved/authored toggles stay comment-less on every future run. Action: Backfill a leading wire comment on every Novita file that has toggle, and make translateModel attach that header whenever the translated model’s reasoning_options include toggle (same pattern as OpenRouter/ai&), not only for the verified-ID allowlist.
[high] [possible mistake] providers/novita-ai/models/deepseek/deepseek-v4-pro.toml:7 - Check: Reasoning effort must match this host’s real controls / lab + same-surface peers, not an invented GPT-style ladder. Why: Final Novita V4 Pro keeps effort = ["low","medium","high","xhigh"] and V4 Flash keeps ["minimal","low","medium","high","xhigh"], while first-party DeepSeek is high/max (Flash also low), and the PR’s own notes say unknown effort must not be published. These files are rewritten here and will be frozen by sync via preserved reasoning_options. Action: Verify Novita’s actual reasoning_effort (or equivalent) for these routes; if unverified, drop effort to lab/peer-safe values or toggle-only after live proof—do not keep the L/M/H/xhigh ladder by default.
[medium] [violation] providers/novita-ai/provider.toml:4 - Check: Provider-level reasoning docs must match the wire syntax the catalog claims. Why: provider.toml still documents top-level enable_thinking for a small GLM/DeepSeek set, while new/updated model headers and VERIFIED_TOGGLE_HEADER claim thinking.type = enabled|disabled. Consumers reading the provider file will use the wrong control. Action: Update the provider.toml reasoning comment block so it documents the verified thinking.type path (and any remaining enable_thinking models, if still accurate), consistent with the model headers.
[medium] [possible mistake] packages/core/src/sync/providers/novita-ai.ts (deleteMissing: true) - Check: Authoritative deletion only when the source catalog is complete for the automation account. Why: The PR enables deleteMissing: true and deletes dozens of local models, while the original PR summary still describes an account-scoped inventory that should retain missing locals. A partial/key-scoped list plus deletion permanently drops still-served models (the 50% guard only stops large shrinks). Action: Confirm with a first-party Novita reference that /openai/v1/models is a full public catalog for NOVITA_API_KEY; if it can be account- or tier-scoped, set deleteMissing: false (or only delete with a stronger completeness signal).

github-actions · 2026-09-18T07:14:27Z

Action items

[high] [violation] providers/novita-ai/models/minimax/minimax-m2.1.toml - Check: Provider overrides must not contradict lab/base_model capability facts without host-specific evidence. Why: The factored entry keeps reasoning = false while base_model = "minimax/MiniMax-M2.1" (lab + first-party MiniMax + OpenRouter peers are reasoning = true, usually with reasoning_options = []). That publishes a false non-reasoner and leaves a dangling [interleaved] side channel on a model marked non-reasoning. Action: Drop the reasoning = false override (inherit lab true), author reasoning_options for this host (peer-style [] unless Novita exposes a real control), and keep/remove interleaved consistently with that choice.
[medium] [violation] packages/core/src/sync/providers/novita-ai.ts (authoritativeHeaders / translateModel header) - Check: authoritativeHeaders must not erase curated leading comments when the translator has nothing to write. Why: With authoritativeHeaders: true, the runner sets header = translatedHeader ?? "". Header is only emitted when the translated model has a toggle; every other model (effort-only, [], non-reasoners, source notes) gets undefined → empty header, so hourly sync wipes existing leading comments. Action: Either stop setting authoritativeHeaders, or always return a deliberate header (including undefined meaning “leave existing”), or only replace headers when translated.header is non-empty.
[medium] [possible mistake] providers/novita-ai/models/meta-llama/llama-3.3-70b-instruct.toml - Check: Synced limits should match the host’s real context/output, not a truncated inventory value. Why: Context/output collapse from 131_072 / 120_000 to 12_288 / 12_288 while the lab default is ~128k; that is a large capability regression if the API field is wrong or incomplete. Action: Confirm against Novita’s model-detail/docs for this ID; if 12k is wrong, restore the real limits (or omit so lab defaults apply) and cite the source in the PR body.

github-actions · 2026-09-18T08:04:09Z

Action items

[high] [violation] packages/core/src/sync/providers/novita-ai.ts / providers/novita-ai/models/deepseek/deepseek-v4-*.toml - Check: Relay reasoning_options must follow lab + same-surface peer controls for the model, not an incomplete set from partial testing. Why: Final sync forces toggle-only for deepseek-v4-flash / deepseek-v4-pro (VERIFIED_TOGGLE_ONLY) and creates deepseek-v4.1-flash, deepseek-v4-flash-0731, and deepseek-v4-flash-vision-exp with only { type = "toggle" }. First-party DeepSeek and established relays expose toggle plus effort (e.g. lab low|high|max / high|max; OpenRouter high|xhigh or low|high|max). Comment text that effort “was not verified” is uncertainty, not affirmative proof that Novita has no effort wire field—publishing toggle-only encodes “no effort control” and will keep re-stripping real effort on future syncs. Action: Live-test Novita reasoning_effort (or the real effort path) for each DeepSeek V4 ID; author the verified effort list with the toggle, or document host-proof that effort is ignored and keep toggle-only only after that proof. Update VERIFIED_TOGGLE_ONLY / create lists so automation cannot regress peer-complete controls.
[high] [possible mistake] packages/core/src/sync/providers/novita-ai.ts (deleteMissing: true) - Check: Deletion is safe only when the remote inventory is the full public catalog, not account-scoped. Why: The PR enables authoritative deletes (plus a 50% shrink guard) and removes a large set of existing Novita TOMLs. The PR body still describes account-scoped inventory retention, while authenticated /v1/models endpoints are commonly visibility-scoped (OpenAI/Merge Gateway keep deleteMissing: false for that reason). A key that sees a subset will permanently drop still-served models once under the fraction guard. Action: Confirm with Novita docs or multi-account inventory that this endpoint is the complete public catalog for every automation key; if not, set deleteMissing: false (or only delete IDs proven retired) and restore any still-served removals.
[medium] [possible mistake] providers/novita-ai/models/qwen/qwen3-max.toml - Check: Provider reasoning / reasoning_options overrides must match this host’s real behavior relative to lab metadata and peers. Why: Lab models/alibaba/qwen3-max.toml and OpenRouter’s qwen3-max entry are non-reasoning; this PR sets reasoning = true, interleaved, and toggle controls. That is a large capability flip on a flagship ID if Novita only accepts thinking.type without actually producing reasoning. Action: Keep the override only with clear host evidence (sample request/response showing reasoning tokens/reasoning_content on and off). Otherwise drop reasoning / reasoning_options / interleaved and leave the model non-reasoning like the lab and peers.
[medium] [possible mistake] providers/novita-ai/models/qwen/qwen3.5-plus.toml (and other new Qwen 3.5/3.6/3.7/3.8 Novita reasoners) - Check: Relay options should be the intersection of lab controls this host actually exposes. Why: Alibaba first-party Qwen 3.5+ reasoners commonly use toggle + budget_tokens (thinking_budget). These Novita files publish toggle-only. That is valid only if Novita does not forward a reasoning budget; if it does, toggle-only understates caller control the same way incomplete DeepSeek effort does. Action: Verify whether Novita accepts thinking_budget / equivalent on these routes; add budget_tokens when present, or note in the leading header that budget is unsupported on this host so future syncs do not look like unfinished ports.

github-actions · 2026-09-18T08:38:30Z

Action items

[high] [violation] packages/core/src/sync/providers/novita-ai.ts - Check: reasoning_options precedence must match the verified DeepSeek V4 control sets the PR claims to publish. Why: Final order still evaluates VERIFIED_THINKING_TOGGLE before the Flash effort branch. deepseek/deepseek-v4-flash-0731 remains in that toggle set, so hourly sync will rewrite the new toggle + effort file back to toggle-only. deepseek/deepseek-v4-flash-vision-exp is also in VERIFIED_THINKING_TOGGLE while VERIFIED_TOGGLE_ONLY and the TOML disagree (toggle-only vs effort). Action: Make each ID appear in exactly one control path. Remove Flash-0731 (and any other effort-verified Flash IDs) from VERIFIED_THINKING_TOGGLE, and either keep vision-exp as verified toggle-only (strip effort from its TOML) or move it onto the same effort list and drop it from both toggle-only sets. Add a unit test that re-translating each of these IDs yields the committed option set.
[high] [violation] providers/novita-ai/models/deepseek/deepseek-v4-flash-0731.toml - Check: TOML must not duplicate reasoning_options or wire comments. Why: Patch 15 inserts the Effort header and [[reasoning_options]] type = "effort" block twice. Consumers see a malformed control surface, and the next sync cannot “heal” this cleanly while the precedence bug above remains. Action: Keep a single # Effort: reasoning_effort = low|high|max comment and a single effort option block with values = ["low", "high", "max"].
[high] [violation] providers/novita-ai/models/qwen/qwen3.6-27b.toml (and sibling qwen3.6-35b-a3b, qwen3.6-plus, qwen3.8-27b, qwen3.8-flash, qwen3.8-max) - Check: Catalog files must match verified host controls for every ID in VERIFIED_BUDGET_TOGGLE. Why: Sync now emits toggle + budget_tokens for these Qwen IDs (and lab/Alibaba peers use that shape), but only qwen3-max.toml and qwen3.5-plus.toml were updated. The other six new entries remain toggle-only without a Budget wire comment, so the committed catalog understates controls the PR says were live-tested. Action: Add [[reasoning_options]] type = "budget_tokens" plus a leading # Budget: thinking_budget (integer reasoning tokens) comment to every VERIFIED_BUDGET_TOGGLE Novita file, or remove those IDs from the budget set if budget was not actually verified on those routes.
[medium] [possible mistake] providers/novita-ai/models/qwen/qwen3-max.toml - Check: Provider reasoning overrides must not contradict lab/first-party facts without host-specific evidence. Why: models/alibaba/qwen3-max.toml and providers/alibaba/models/qwen3-max.toml both set reasoning = false, while this PR publishes Novita reasoning = true with toggle + budget. That is only valid if Novita’s route truly enables thinking for this ID. Action: Confirm live Novita chat/completions behavior for qwen/qwen3-max (thinking on/off and thinking_budget), cite that evidence in the PR body, or drop the reasoning override and keep it non-reasoning like the lab entry.

github-actions · 2026-09-18T10:50:54Z

Action items

[high] [possible mistake] providers/novita-ai/models/qwen/qwen3-max.toml:1 - Check: Host reasoning must match this API’s real controls and not invent a reasoner when lab/peers do not. Why: Lab models/alibaba/qwen3-max.toml and first-party providers/alibaba/models/qwen3-max.toml both set reasoning = false; OpenRouter also factors it as non-reasoning. This PR forces reasoning = true plus toggle + budget_tokens and keeps that in VERIFIED_BUDGET_TOGGLE. That can mislabel a non-thinking Max route (or confuse it with a thinking variant). Action: Confirm live Novita chat/completions for qwen/qwen3-max actually returns reasoning tokens under thinking.type / thinking_budget. If not, drop the reasoning override and remove it from VERIFIED_BUDGET_TOGGLE. If yes, keep the override and cite the host-specific evidence in the PR body.
[high] [possible mistake] providers/novita-ai/models/deepseek/deepseek-v4-flash-vision-exp.toml:1 - Check: Relay DeepSeek V4 controls should follow lab/same-surface peers unless this host is proven narrower. Why: First-party providers/deepseek/models/deepseek-v4-flash-vision-exp.toml uses toggle + effort low|high|max. Novita V4 Flash / V4.1 Flash siblings in this PR use the same effort set, but vision-exp is hard-coded toggle-only via VERIFIED_TOGGLE_ONLY and the tests lock that in. Action: Re-test reasoning_effort on Novita for this ID. If effort works, publish toggle + ["low","high","max"] (and the Effort wire comment). If effort is ignored, keep toggle-only and document that host-specific finding next to the model/sync allowlist.
[medium] [violation] packages/core/src/sync/providers/novita-ai.ts - Check: Leading wire comments must document every published control (toggle, and effort/budget when present). Why: translateModel only emits the toggle header (plus a budget line for Qwen). DeepSeek V4 effort routes still get effort in reasoning_options, but the generated header never includes # Effort: reasoning_effort = …. Hand-edited files currently carry those comments, but creates/re-syncs will not. Action: Build the header from the actual option set (toggle + effort values and/or budget), matching the TOML comments already present on the V4 Flash/Pro files.
[low] [possible mistake] packages/core/src/sync/providers/novita-ai.ts - Check: Provider flags should match the deletion policy they claim. Why: Final config sets deleteMissing: false (correct for account-scoped inventory) but still sets maxMissingFraction: 0.5. The runner only evaluates that guard when deletions are enabled, so the shrink guard never runs. Action: Remove the unused maxMissingFraction (and any tests that only exist for it), or re-enable deletions only if the endpoint is proven complete for this key.

github-actions · 2026-09-18T11:12:04Z

Action items

[high] [violation] providers/novita-ai/models/qwen/qwen3.5-27b.toml:1 - Check: Reasoning options must follow lab + same-surface peers (AGENTS.md / audit skill). Why: These Novita Qwen 3.5/3.7 routes are factored onto Alibaba lab models whose first-party entries use toggle + budget_tokens, and this PR already publishes that pair for qwen3.5-plus / qwen3.6* / qwen3.8* / qwen3-max after live thinking_budget checks. The same PR only authors toggle (plus a wire header) for qwen3.5-27b, qwen3.5-35b-a3b, qwen3.5-122b-a10b, qwen3.5-397b-a17b, and qwen3.7-max, and the sync allowlist never maps them to budget. That understates caller controls relative to lab/peers on the same host. Action: Either add verified budget_tokens (+ budget wire comment) for those IDs in the TOMLs and VERIFIED_BUDGET_TOGGLE, or document live proof that Novita rejects/ignores thinking_budget on each ID and keep toggle-only deliberately.
[high] [possible mistake] packages/core/src/sync/providers/novita-ai.ts (final deleteMissing: false) / deleted providers/novita-ai/models/** - Check: Deletions must be authoritative for the served catalog (sync.md). Why: The PR body and final Novita notes say the authenticated inventory can be account/tier-scoped and must not drive removals, and the sync module ends with deleteMissing: false. The same PR still deletes dozens of existing Novita model files (Baidu, DeepSeek, Qwen, Llama, Sao10K, Xiaomi, ZAI, etc.). If those IDs are only invisible to the automation key, the catalog permanently loses still-served models. Action: Restore any IDs still offered publicly (or still intended in the catalog), and cite a non-account-scoped source for each intentional removal; do not ship bulk deletes from a scoped key response.
[medium] [violation] providers/novita-ai/models/qwen/qwen3.5-27b.toml (and peer Qwen 3.5/3.7 + several GLM/DeepSeek toggle files) - Check: Toggle reasoners on this host should expose the verified reasoning side channel. Why: Headers claim live checks where disabling thinking.type removes reasoning_content, and many new/updated routes correctly set [interleaved] field = "reasoning_content". Several toggle-only updates omit interleaved entirely (e.g. qwen3.5-27b, qwen3.5-35b-a3b, qwen3.5-122b-a10b, qwen3.5-397b-a17b, qwen3.7-max, zai-org/glm-4.7-flash, deepseek/deepseek-v3.1), so clients cannot discover the side channel the verification depends on. Action: Add interleaved = { field = "reasoning_content" } wherever the toggle was verified to emit reasoning_content, and make the sync path set that for those IDs on create/update.
[medium] [violation] providers/novita-ai/models/deepseek/deepseek-v3.2-exp.toml:1 - Check: Non-lab hosts must use base_model when the lab model is nameable (AGENTS.md). Why: Novita did not create DeepSeek V3.2 Exp; the file remains a full inline lab definition while sibling DeepSeek routes were factored. There is no models/deepseek/deepseek-v3.2-exp.toml on the base revision. Action: Add a complete models/deepseek/deepseek-v3.2-exp.toml lab entry and reduce the Novita file to override-only (cost, reasoning_options, interleaved, real deltas). Apply the same pattern to any other still-inline third-party Novita DeepSeek routes (e.g. deepseek-v3.1-terminus) that are nameable lab models.
[low] [possible mistake] providers/novita-ai/models/minimax/minimax-m2.1.toml:1 - Check: Provider overrides must not invent inconsistent reasoning metadata. Why: The final file drops the prior reasoning = false override, keeps [interleaved], and sets reasoning_options = [] while base_model = "minimax/MiniMax-M2.1" inherits reasoning = true. That is coherent only if Novita always-on reasons with no caller control; it is wrong if the route is non-reasoning or toggleable. Action: Confirm Novita’s live behavior for minimax/minimax-m2.1 and either keep an explicit non-reasoning override, always-on [], or a verified toggle—not a mix of interleaved + ambiguous inheritance.

Alex-yang00 added 12 commits September 14, 2026 18:07

feat(sync): add Novita AI model catalog sync

e63bf80

fix(sync): accept Novita model list response

b3eb982

feat(sync): map Novita catalog metadata

ffde31b

feat(sync): automatically sync Novita model metadata

010bff0

feat(sync): enable full Novita catalog lifecycle

a43f094

fix(sync): avoid guessing Novita pricing and model capabilities

5fbd74a

fix(sync): recognize Novita free models and verified controls

89c2339

fix(sync): harden Novita catalog automation and preserve authored prices

ed34f5b

feat(novita-ai): sync DeepSeek V4.1 Flash with verified reasoning toggle

3dc8eeb

feat(novita-ai): sync models with verified thinking controls

d9941f7

merge dev and preserve Novita sync hardening

f136248

fix(novita-ai): tolerate zero context catalog entries

0ab93c0

Alex-yang00 marked this pull request as ready for review September 18, 2026 05:37

fix(novita-ai): address automated review findings

318642d

fix(novita-ai): resolve reasoning review findings

a6acdf7

fix(novita-ai): address latest review findings

ffaa945

fix(novita-ai): align sync controls with live API behavior

53d8701

fix(novita-ai): preserve verified reasoning controls

6552543

fix(novita-ai): complete verified reasoning metadata

8b0d622

Navigation Menu

feat(sync): add Novita AI model catalog sync - #7050

feat(sync): add Novita AI model catalog sync#7050
Alex-yang00 wants to merge 18 commits into
anomalyco:devfrom
Alex-yang00:feat/novita-ai-model-sync

Alex-yang00 commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Alex-yang00 commented Sep 14, 2026

Summary

Data source

Required repository secret

Validation

Notes

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

github-actions Bot commented Sep 18, 2026

Action items

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant